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Abstract 

j 

Three  questions  were  addressed  in  an  experiment  in  which 
subjects  followed  instructions  to  complete  tasks  involving  several 
pieces  of  electronic  equipment:  (1)  Two  instruction  formats  were 
compared:  a  hierarchical  menu  format  containing  natural  chunks  of 

instructions  was  not  superior  overall  to  a  simple  step-by-step 
instruction  format .  The  menu  format  was  superior  only  if  the 
subject  was  familiar  with  the  type  of  device,  and  was  sometimes 
substantially  inferior  otherwise.  (2)  Experts  were  compared  to 
nonexperts,  and  found  to  be  faster  overall,  and  able  to  operate 
equipment  with  fewer  instructions  in  the  menu  condition.  They 
were  also  faster  when  complex  physical  actions  were  involved. 
Thus,  there  were  both  specific  and  general  effects  of  expertise. 
(3)  Evidence  was  sought  that  knowledge  of  how  to  operate  equipment 
was  schematic.  It  was  expected  that  when  subjects  in  the  menu 
format  condition  operated  a  device  without  selecting  any 
instructions  to  read,  their  sequence  of  actions  should  correspond 
to  stereotyped  schema-like  patterns.  This  occurred  only  weakly, 
suggesting  that  even  experts  operate  everyday  devices  in  a 
problem-solving  mode,  rather  than  by  retrieved  complete 
procedures . 
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How  Experts  and  Nonexperts  Operate 
Electronic  Equipment  from  Instructions 


David  E.  Kieras,  Mark  Titbits, 
and 

Susan  Bovair 


This  report  describes  results  from  an  experiment  which  was 
designed  to  assess  three  questions  about  how  people  operate  a 
piece  of  equipment  from  written  instructions .  The  questions  deal 
with  instruction  format,  expertise,  and  the  organization  of  prior 
knowledge,  in  a  task  in  which  subjects  must  follow  a  set  of 
instructions  in  order  to  complete  a  task  involving  an  electronic 
device . 

The  first  question  is  one  of  instruction  format  (see  Smith  & 
Goodman,  1982).  This  is  the  difference  between  whether  the  format 
or  layout  of  the  instructional  material  forces  the  user  to  execute 
each  step  in  order,  or  whether  the  instructions  allow  the  user  to 
pick  and  choose  the  material  to  be  read  and  executed.  In  this 
experiment,  one  group  received  step-by-step  instructions  that  were 
presented  a  single  step  at  a  time,  and  the  subject  had  to  read 
every  step.  The  other  group  received  a  hierarchical  menu  of 
instructions,  in  which  the  subject  could  either  execute  the  task 
with  only  a  high  level  description,  or  could  request  more  detail. 
In  this  way,  the  subject  would  only  have  to  read  the  instructions 
that  he  or  she  felt  was  necessary  to  execute  the  task.  The 
rationale  of  this  manipulation  is  that  an  expert  subject  could 
take  advantage  of  the  hierarchical  menu  format,  because  large 
portions  of  the  task  would  be  familiar.  However,  a  nonexpert 
subject  would  have  to  read  all  of  the  instructions  anyway,  so  the 
menu  would  not  be  of  any  great  advantage.  Furthermore,  there 
should  be  relatively  little  difference  between  experts  and 
nonexperts  on  step-by-step  instructions,  because  in  both  cases  all 
of  the  steps  must  be  read. 

The  second  question  is  the  nature  of  expertise  effects. 
While  expertise  has  been  heavily  studied  (see  Chi,  Feltovich,  & 
Glaser,  1981;  Chi  &  Glaser,  in  press),  it  has  not  been  examined 
in  the  context  of  operating  electronic  equipment,  a  domain  of 
great  practical  importance.  Generally,  it  is  expected  that 
experts  would  complete  the  tasks  faster,  and  read  fewer  steps  in 
the  menu  condition.  However,  this  could  depend  on  the  device 
under  consideration.  Only  experts  would  be  familiar  with  some 
devices,  but  even  the  nonexperts  should  be  able  to  operate  other 
devices  easily.  Likewise,  even  nonexperts  should  know  some  things 
about  almost  any  device,  such  as  how  to  turn  it  on.  Thus,  it  was 
expected  that  there  would  be  an  interaction  of  subject  expertise, 
experience  with  the  exact  device,  and  the  nature  of  particular 
steps  in  the  instructions.  The  basic  question  about  expertise 
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effects  is  whether  there  are  general  effects  of  expertise,  or 
whether  they  are  specific  to  the  individual  devices  involved.  For 
this  reason,  several  devices  of  widely  differing  familiarity  were 
used  • 


The  third  question  concerns  the  nature  of  the  prior  knowledge 
that  subjects  have  about  devices.  In  Kieras  ( 1 982")  it  was 
suggested  that  knowledge  of  devices  is  organized  in  the  form  of 
schemas.  These  schemas  would  include  knowledge  not  only  of  how  to 
recognize  a  particular  type  of  device,  but  also  its  typical 
structure  and  operating  procedures.  If  device  knowledge  is 
organized  by  schemas,  there  should  be  clear  patterns  in  the  data 
obtained  in  this  experiment.  Menu  choices  should  follow  patterns 
that  would  be  expected  from  schematic  device  knowledge.  If 
subjects  operated  the  equipment  entirely  from  prior  knowledge, 
without  reading  instructions,  which  happened  in  many  cases,  then 
their  behavior  should  follow  some  pattern  that  can  be  described  in 
terms  of  device  schemas. 

The  basic  manipulations  performed  in  this  study  were  as 
follows:  several  devices  were  used,  which  included  two  every-day 
devices,  two  devices  familiar  to  only  experts,  and  two  novel 
devices  familiar  to  neither  experts  nor  nonexperts.  The  subjects 
were  either  experts,  who  typically  had  several  years  of  working 
experience  in  electronics,  or  nonexperts,  who  were  ordinary 
college  students.  A  questionnaire  was  used  to  confirm  the 
subject's  classification,  and  to  assess  each  subject's  experience 
with  the  individual  devices  used  in  the  experiment.  The  two 
instruction  formats  were  either  a  step-by-step  format  or  a 
hierarchical  menu  format.  The  terminal  nodes  of  the  menu 
hierarchy  consisted  of  the  exact  same  individual  instruction  steps 
as  were  used  in  the  step-by-step  format.  The  variables  measured 
were  the  total  completion  time  for  each  task  on  each  device,  the 
completion  time  for  each  individual  step  in  the  step-by-step 
instructions,  and  in  the  menu  condition,  the  individual  menu 
choices,  and  their  completion  times.  The  subjects'  behavior  was 
recorded  on  videotape  to  allow  detailed  scoring  on  the  subjects' 
activities  while  performing  the  tasks. 


METHOD 


Materials 


Devices .  The  six  devices  used  are  described  in  Table  1 .  The 
radio,  cassette  recorder,  VOM,  and  oscilloscope  were  of  a  standard 
make.  The  phi  phenomenon  demonstrator  was  professionally  built, 
but  in  general  construction  style  it  appeared  to  be  a  "home-brew" 
amateur  job.  The  physiological  stimulator  is  a  standard  piece  of 
apparatus  in  a  physiological  psychology  lab,  but  as  the  ratings 
confirmed,  it  was  essentially  unfamiliar  to  all  subjects.  Notice 
that  all  of  the  non-everyday  devices  were  relatively 
oid-fashioned ,  being  from  the  vacuum-tube  era.  The  devices  were 
prepared  before  presentation  to  each  subject  by  setting  all 
controls  to  incorrect  positions  so  that  in  order  to  complete  the 


Table  1 

Devices  Used  in  the  Experiment 


Device 

Description 

1 .  Rad i o 

A  portable  AM-FM  radio,  with  built-in  AC 
adapter,  antenna,  volume,  tone,  tuning, 
and  band  controls. 

2.  Recorder 

A  portable  audio  cassette  tape  recorder, 
with  keyboard  tape  controls,  red  record 
interlock  key,  and  volume  control.  Supplied 
cassette  was  not  fully  rewound. 

3-  V  CM 

A  standard  volt-ohm-milliameter ,  with  a 
supplied  resistor  to  measure. 

4.  Oscilloscope 

A  dual-trace  triggered-sweep  oscilloscope 
with  standard  audio  signal  generator  and 
connecting  cables. 

5*  Phi  Phenomenon 
Demonstrator 

A  device  that  flashes  two  connected  neon 
bulbs  alternately  at  various  rates  and  phase 
relationships . 

6.  Physiological 
Stimulator 

A  large  device  with  several  dial-multiplier 
sets  that  produce  pulses  of  specified 
magnitude  rate,  and  duty  cycles;  a  neon 
bulb  is  connected  to  the  output  to  indicate 
the  pulses. 
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task,  each  control  would  have  to  be  properly  set. 

Instructions .  A  major  goal  in  composing  the  instructions  was 
to  allow  the  menu  and  the  step-by-step  instructions  to  be  easily 
compared  to  each  other.  This  was  done  by  preparing  the  materials 
so  that  the  terminal  steps  in  the  menu  instructions  were  exactly 
identical  to  the  steps  comprising  the  step-by-step  instructions, 
and  were  worded  and  displayed  identically. 

The  menu  instructions  made  up  a  hierarchy  of  natural  "chunks" 
of  the  operating  procedure.  Determination  of  the  chunks  was  done 
intuitively.  It  is  clear  from  some  aspects  of  the  results  that 
some  of  the  chunks  chosen  were  in  fact  natural  units;  however, 
the  data  do  not  definitively  confirm  the  chunk  classification. 

Each  set  of  instructions  began  with  a  statement  of  the  task 
that  the  subject  had  to  accomplish.  This  main  task  statement  was 
specific  enough  that  the  subject  could,  if  he  or"  she  had  adequate 
prior  knowledge,  complete  the  entire  task  from  just  this 
statement.  However,  the  main  task  statement  did  not  describe  how 
the  controls  on  the  device  had  to  be  set  or  operated.  Table  2 
lists  the  tasks  that  were  to  be  performed  on  each  device,  in  the 
same  wording  as  they  were  shown  to  subjects. 

Subjects 

The  nonexperts  were  recruited  by  campus  and  newspaper 
advertisements,  and  were  paid  $5  for  participating.  As  shown  by 
the  experience  questionnaires  administered  to  the  subjects,  only 
one  expert  subject  was  inadvertently  recruited  by  this  method. 
The  expert  subjects  were  recruited  by  advertisements  directed  at 
electronics  experts.  In  all  cases,  the  subjects  obtained  were 
highly  experienced  in  electronics;  the  typical  expert  had  several 
years  experience  as  an  electronics  technician  in  the  military. 
Twenty  subjects  were  recruited  by  each  method,  but  in  the  analyses 
used  below,  the  classification  was  corrected,  to  yield  nineteen 
nonexperts  and  twenty-one  experts.  Since  earlier  studies  seemed 
to  suggest  that  there  were  strong  sex  differences  among 
nonexperts,  and  female  electronics  experts  were  extremely  hard  to 
locate,  all  subjects  used  in  this  experiment  were  male. 

Design .  The  instruction  format  condition  was  determined  at 
random  for  each  subject.  Each  subject  carried  out  the  six  tasks 
on  the  six  devices  in  the  same  instruction  format  condition.  The 
device  tasks  were  done  in  a  fixed  order,  which  is  the  order  in 
which  the  devices  are  listed  in  Table  1 .  This  order  was  chosen  to 
present  the  tasks  and  devices  in  order  of  decreasing  familiarity, 
and  increasing  apparent  complexity  within  each  level  of 
familiarity. 


Table  2 


Task  Main  Task  Statement 


1 .  Listen  to  Station  KUAT-PM  (90.5  PM)  at  medium  volume  on 
the  portable  radio. 

2.  Record  the  words  "testing.  .  .  1,  2,  3"  on  the  cassette 
recorder,  and  play  the  words  back  at  medium  volume. 

3.  Measure  the  resistance  of  the  resistor  using  the  volt-ohm 
meter . 

4.  Use  the  signal  generator  and  the  oscilloscope  to  display 
about  two  AC  wave  cycles  on  the  oscilloscope  screen. 

5.  Use  the  phi  phenomenon  demonstrator  to  flash  the  lights 
at  5  CPS  (cycles  per  second). 

6.  Use  the  stimulator  to  flash  the  neon  light  at  a  frequency 
of  1  CPS  (cycles  per  second)  with  a  flash  duration  of 

.7  seconds  and  a  delay  of  .5  seconds. 
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Apparatus  and  Procedure 

Each  subject  was  run  individually,  and  was  seated  in  a  small 
room  before  a  table.  On  the  right-hand  end  of  the  table  was  a 
standard  video  terminal,  on  which  a  laboratory  computer  displayed 
the  instructions.  The  left-hand  portion  of  the  table  was  occupied 
by  the  device.  A  videotape  recorder  recorded  all  of  the  subject's 
activity.  The  instructions  were  presented  one  step  or  menu  at  a 
time  with  the  subject  tapping  the  space  bar  or  typing  a  choice 
number  to  go  on  to  the  next  display.  The  laboratory  computer 
recorded  the  amount  of  time  that  the  subject  left  each  instruction 
step  or  menu  on  the  screen.  Due  to  the  nature  of  the  equipment, 
and  the  prohibitive  scoring  effort  involved,  it  was  not  practical 
to  distinguish  the  time  the  subject  spent  reading  from  the  time 
the  subject  spent  carrying  out  the  instructions.  Thus,  the 
laboratory  computer  was  able  only  to  record  5  completion  time 
for  each  step,  defined  as  the  total  reading  p  3  execution  time 
for  the  instruction  step.  The  videotape  r  -'rding  was  used  to 
determine  what  subjects  actually  did  on  each  s  -o. 

The  devices  were  brought  into  the  room  or.  ;  a  time,  and  the 
subject  then  carried  out  the  task  on  the  device  When  the  subject 
had  reached  the  end  of  the  instructions,  the  experimenter  returned 
and  checked  that  the  task  had  been  carried  out  correctly,  in  terms 
of  whether  the  final  correct  result  was  achieved.  The  device  was 
then  removed,  and  a  new  device  brought  in.  Subjects  who  did  not 
achieve  the  proper  final  result  were  asked  to  repeat  the  task; 
however,  the  data  from  these  repeated  tasks  were  later  dropped 
from  the  analysis. 

Due  to  inadequate  training  of  the  experimenters,  on  some 
trials  the  equipment  was  being  moved  in  and  out  of  the  room  while 
the  clock  was  running,  making  the  completion  time  record  of  the 
first  instruction  unreliable.  It  is  believed  that  these  events 
are  not  confounded  with  any  of  the  experimental  manipulations,  so 
the  analysis  of  total  completion  time  would  be  conservative  due  to 
the  extraneous  variability.  Examination  of  the  video  tapes  shows 
that  the  subjects  were  visually  inspecting  the  devices  while  they 
were  being  brought  in,  and  so  these  times  reflect  the  total  time 
that  the  subjects  interacted  with  the  devices  to  complete  the 
task. 


RESULTS 


Total  Completion  Times 

Analysis  method .  The  total  completion  time  for  each  subject 
on  each  task  was  calculated  as  the  total  elapsed  time  from  the 
presentation  of  the  main  task  statement  until  the  experimenter  had 
confirmed  that  the  task  was  completed  correctly.  Data  from  tasks 
were  dropped  in  which  the  subject  did  the  entire  task  more  than 
once,  or  failed  to  do  the  task  at  all  correctly.  Out  of  the  total 
of  240  task  attempts,  14  were  thus  dropped.  Due  to  the  unequal 
group  sizes,  missing  data,  and  unbalanced  device  experience 
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factor,  the  total  times  were  analyzed  using  stepwise  multiple 
regression . 

The  subject's  expertise  group,  instruction  format  condition, 
and  subject's  experience  with  the  individual  device  were 
represented  as  dummy  variables.  The  device  experience  variable 
was  based  on  the  questionnaires  that  each  subject  filled  out.  If 
the  subject  indicated  any  actual  usage  experience  with  the  device, 
then  the  device  experience  dummy  variable  received  a  value  of  one; 
otherwise  a  value  of  zero  was  assigned.  The  device  factor  was 
represented  as  a  set  of  five  dummy  coded  variables  with  the  radio 
being  used  as  the  baseline.  Following  the  method  suggested  by 
Pedhazur  (1982)  for  mixed  designs,  a  variable  whose  value  is  the 
subject's  mean  total  completion  time  over  the  six  devices  was 
included.  The  between-sub j ects  factors  and  interactions  were 
entered  first  in  the  equation,  followed  by  the  subject's  mean  time 
variable,  followed  by  all  of  the  within-sub j ect  factors  and 
interactions.  The  analysis  was  hierarchical,  in  that  main  effects 
were  forced  into  the  equation  before  interactions. 

All  of  the  interactions  between  subject  experience,  device 
experience,  and  instruction  format  condition  were  represented,  but 
only  instruction  format  condition  and  subject  expertise  group  were 
allowed  to  interact  with  the  device  factor;  device  experience  was 
not  allowed  to  interact  with  the  device  factor.  The  rationale  for 
this  decision  is  that  the  device  experience  variable  is  already 
specific  to  individual  devices,  so  interactions  between  individual 
device  experience  and  individual  device  dummy  variables  would  be 
difficult  to  interpret. 

Note  that  subject  expertise  and  specific  device  experience  in 
these  data  are  only  slightly  correlated  (£=.13) >  and  the 
interaction  between  subject  expertise  and  device  experience  was 
not  significant.  Thus  these  two  factors  make  practically 
independent  contributions  to  the  total  completion  times.  Two  of 
the  devices  were  familiar  to  everyone,  and  two  were  unfamiliar  to 
almost  everyone,  resulting  in  these  two  variables  being  nearly 
orthogonal . 

With  a  total  of  23  variables  in  the  equation  and  163  degrees 
of  freedom  in  the  residual,  81.5$  of  the  variance  in  the  total 
completion  times  was  accounted  for.  This  extremely  high  figure  is 
due  to  two  factors:  the  subject's  mean  completion  time  accounted 
for  approximately  15$  of  the  variance,  and  the  device  factor 
accounted  for  about  50$  of  tne  variance.  This  is  clearly  due  to 
the  fact  that  the  devices  varied  substantially  in  number  of  steps 
in  the  tasks,  and  thus  the  completion  times  vary  systematically 
over  an  extremely  wide  range.  The  effects  to  be  discussed  below 
were  all  tested  for  significance  at  the  .05  level,  using  the 
" F-to-remove"  statistic,  which  is  a  conservative  estimate  of  the 
significance  of  an  individual  variable  as  if  it  were  the  last  to 
enter  the  equation. 
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Main  effects .  Table  3  shows  the  means  for  the  vai  ious  main 
effects  that  were  significant.  The  subject  expertise  variable  was 
quite  significant;  experts  were  about  one  third  faster  in 
completion  time  than  nonexperts.  There  was  no  significant  main 
effect  of  instruction  format  condition,  even  though  the  menu 
condition  averaged  about  30  seconds  faster.  This  means  that, 
counter  to  intuition,  the  menu  format  was  not  reliably  superior 
overall  to  the  step-by-step  format.  This  is  probably  a  result  of 
the  fact  that  while  fewer  steps  were  read  in  the  menu  condition, 
more  material  has  to  be  read  in  addition  to  the  individual  steps. 
The  device  experience  factor  was  significant;  being  familiar  with 
a  specific  device  led  to  a  30#  improvement  in  completion  time.  As 
would  be  expected,  there  is  a  very  strong  main  effect  of  devices. 

Interactions .  The  interaction  between  device  experience  and 
instruction  format  condition,  shown  in  Table  4,  was  significant. 
The  menu  instructions  are  actually  slower  than  the  step-by-step 
instructions  if  the  device  is  not  familiar,  but  substantially 
faster  than  the  step-by-step  instructions  if  the  device  is 
familiar.  Thus,  not  only  do  the  menu  instructions  allow  the  user 
to  take  advantage  of  prior  knowledge  more  than  the  step-by-step 
instructions,  but  the  lack  of  prior  knowledge  means  that  the  extra 
"overhead"  in  menu  instructi ons ,  plus  mistakes  made  as  a  result  of 
skipping  instructions ,  actually  slows  down  task  completion. 

The  interaction  of  instruction  format  condition  and  device, 
whose  means  are  shown  in  Table  5»  was  significant.  For  the  radio, 
recorder,  and  phi  demonstrator,  the  menu  condition  produced  faster 
results  than  the  step-by-step  condition.  However,  the  VOM, 
oscilloscope,  and  stimulator  produced  the  opposite  effect.  This 
is  probably  due  to  the  fact  that  these  are  devices  which  were 
especially  difficult  for  nonexperts,  exaggerating  the  effect  of 
the  extra  material  in  the  menu  format.  Table  6  shows  the 
interaction  between  devices  and  subject  expertise  group,  which  was 
also  significant.  Here  it  is  clear  that  the  oscilloscope  and  VOM 
were  especially  hard  for  the  nonexperts  compared  to  the  experts. 

The  three-way  interaction  between  subject  expertise, 
condition,  and  device  was  significant,  and  illustrates  the  key 
result.  The  means  are  shown  in  Table  7,  which  includes  the 
percent  gain  resulting  from  using  the  menu  instructions  instead  of 
step-by-step,  for  nonexperts  and  experts  on  each  device.  One 
clear  result  is  that  the  experts  benefit  from  the  use  of  the  menu 
format  on  all  devices  except  for  the  stimulator,  where  there  is  a 
substantial  impairment  in  performance.  This  is  probably  due  to 
the  fact  that  since  this  was  a  complex  and  novel  device,  the 
experts'  attempts  to  operate  it  without  reading  much  of  the 
instructions  often  led  them  down  "garden  paths."  For  example,  one 
expert  plugged  indicator  light  into  the  wrong  jack,  and  then 
spending  a  long  time  trying  to  set  the  controls  to  light  it.  With 
the  nonexperts,  the  two  expert-familiar  devices,  the  VOM  and  the 
oscilloscope,  produced  much  longer  completion  times  in  the  menu 
instructions  compared  to  the  step-by-step.  Since  many  of  the 
nonexpert  subjects  claimed  experience  in  using  the  VOM,  their 
longer  completion  times  in  the  menu  condition  may  be  similar  to 


Table  3 

Main  Effects  in  Total  Time  Data 


Effect 

Means 

Significance 

Expertise 

Nonexperts 

314.0 

** 

Experts 

214.4 

Device  Experience 

Non-familiar 

319.4 

** 

Familiar 

223.0 

Instruction  Format 

Step-by-step 

274.4 

NS 

Menu 

247.1 

Devices 

Radio 

137.1 

** 

Recorder 

166.2 

VOM 

256.8 

Oscilloscope 

511  -9 

Phi  Demonstrator 

118.8 

Stimulator 

343-3 

Table  4 

Total  Time  (Secs)  as  a  Function 
of  Device  Expertise  and  Instruction  Format 


Instruction 

Format 

Device  Experience 

Step-by-step 

Menu 

Not  Familiar 

307.8 

331 .0 

Familiar 

254.4 

185-1 

Table  5 

Total  Time  (Secs)  as  a  Function 
of  Instruction  Format  and  Device 

Format 

Radio 

Device 

Recrdr  VOM 

Oscil 

PhiDem 

Stim 

Step-by-step 

185-7 

201  -3 

230.1 

504.3 

210.5 

314.4 

Menu 

88.4 

131 .0 

297.8 

521  .9 

167.0 

375-5 

Table  6 

Total  Time 

(secs)  as 

a  Function 

of 

Sub j  ect 

Expertise 

and  Device 

Device 


Expertise 

Radio 

Recrdr 

~rm~~ 

Oscil 

PhiDem 

Stim 

Nonexpert 

165-8 

188.7 

342.8 

649-0 

230.3 

367.8 

Expert 

111.0 

145-7 

175.8 

396.4 

151  -2 

321  .3 

Table  7 

Mean  Total  Time  for  each  Device, 
Instruction  Format,  and  Expertise  Group 


Radio 

Recrdr 

Device 

TOM 

Oscil 

PhiDem 

Stim 

Nonexpert 

Step-by-step 

Menu 

224-3 

100.9 

216.3 

158.2 

270.8 

462.8 

604-3 

723-6 

260.4 

196.9 

359-5 

378.0 

#  Gain 

55# 

27# 

-71# 

-20# 

24# 

5# 

Experts 

Step-by-step 

Menu 

147.0 
IQ. 3 

186. 4 
108.8 

189.5 

156.3 

•■'d- 

O  CD 

1  60 . 6 
142.6 

269-2 

373-5 

#  Gain 

47# 

41# 

17# 

4# 

11# 

-39# 
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the  ''garden  path"  effect  obtained  for  the  experts  with  the 
stimulator.  Namely,  a  little  familiarity  with  a  device  is  a 
dangerous  thing;  it  can  lead  to  longer  completion  times  if 
instructions  are  not  followed.  The  elevated  time  for  these 
subjects  in  the  menu  condition  with  the  oscilloscope  is  harder  to 
explain . 

Conclusion .  These  results  demonstrate  that  the  virtues  of 
the  two  instruction  formats  are  heavily  dependent  upon  the  user's 
general  expertise  and  also  the  familiarity  with  the  specific 
device.  In  general,  the  interactions  seem  to  be  due  mostly  to  the 
specific  familiarity  with  the  device,  as  opposed  to  the  subject's 
general  expertise.  That  is,  the  fact  that  the  interaction  between 
device  experience  and  instruction  format  was  significant,  but  the 
interaction  between  subject  expertise  and  instruction  format  was 
not,  suggests  that  the  advantage  of  menu  instructions  is  a  matter 
of  specific  familiarity  with  the  device,  and  not  general 
expertise.  Electronic  experts  may  not  do  better  with  the  menu 
instruction  format  unless  they  have  specific  familiarity  with  the 
device  in  question.  Alternatively,  if  the  device  is  unfamiliar, 
experts  can  benefit  from  menu  instructions  if  the  device  is 
simple,  such  as  the  phi  demonstrator,  but  not  if  it  is  complex, 
such  as  the  stimulator. 

On  the  other  hand,  the  significant  main  effect  of  subject 
expertise,  even  with  specific  device  experience  taken  into 
account,  is  important.  Experts  were  generally  faster  than 
nonexperts  at  operating  the  equipment,  regardless  of  its 
familiarity.  Other  aspects  of  the  results  suggest  that  this  is 
due  not  just  to  faster  execution  of  actions,  and  also  to  better 
organized  and  more  efficient  actions  as  well. 

Menu  Choices 


Number  of  frames  read .  Table  8  shows  the  mean  number  of 
frames  fd'i splays  oT  instruction  steps  or  menus)  read  in  the  menu 
condition  for  each  group  and  each  device.  For  example,  both 
experts  and  nonexperts  read  only  one  frame  for  the  radio,  namely 
the  frame  that  contains  the  main  task  statement,  but  nonexperts 
chose  to  read  an  average  of  50.4  frames  of  information  for  the 
oscilloscope  task,  while  experts  read  an  average  of  only  1.2. 
These  data  were  subjected  to  a  multiple  regression  analysis 
similar  to  the  above  one,  with  the  factors  being  subject 
expertise,  device  experience,  and  devices,  and  interactions  of 
subject  expertise  with  device  experience  and  individual  devices 
were  allowed.  The  results  are  summarized  in  Table  9« 

There  were  strong  main  effects  of  device,  with  the  VOM, 
oscilloscope,  and  stimulator  requiring  many  more  frames  than  the 
radio,  which  was  taken  as  the  baseline.  The  key  results  were  that 
neither  subject  expertise  nor  device  experience,  nor  their 
interaction,  were  significant  predictors  of  the  number  of  frames 
read,  once  the  main  effects  of  device  and  the  interaction  of 
subject  expertise  with  device  were  taken  into  account.  As  shown 
in  Table  8,  the  VOM,  oscilloscope,  phi  demonstrator,  and 


Table  8 

Mean  Number  of  Frames  Read  in  the  Menu  Condition 
for  Each  Expertise  Group 


Group 

Radio 

Recrdr 

VOM 

Oscil 

PhiDera 

Stra 

Nonexperts 

1  .0 

1  .0 

32.5 

50.4 

18.2 

47.8 

Experts 

1  .0 

1  .0 

3-1 

1  .2 

3-4 

21  .9 

Table  9 

Regression  Analysis  on  Number  of 
Frames  Read  in  the  Menu  Condition 


Variable 

Coefficient 

Std  .  Coef. 

F-to-Remove 

CONSTANT 

SUBJECT  EXP. 

10.1 

0.0 

.0 

0.00 

DEVICE  EXP. 

-9.1 

-.231 

3-99 

DEVICE  2 

0. 

-.0 

0. 

DEVICE  3 

29-5 

•  516 

45-40 

DEVICE  4 

46.5 

.839 

113-19 

DEVICE  5 

9-1 

.184 

2.49 

DEVICE  6 

38.7 

•  751 

43-98 

SUB  EXP  X  DEV2 

0.0 

.0 

0.00 

SUB  EXP  X  DEV3 

-26.3 

-.360 

20.82 

SUB  EXP  X  DEV4 

-45-3 

-.618 

61  .52 

SUB  EXP  X  DEV5 

-14.8 

-.224 

8.68 

SUB  EXP  X  DEV6 

-25.9 

-.373 

25.15 

Notes 

R2  is  .85  with  12  variables  and  N=107*  Device  1  (Radio)  is 
used  as  the  baseline  for  dummy  coding  of  Device  factor. 
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stimulator  all  required  many  fewer  frames  for  experts  than  for 
nonexperts.  The  main  effect  of  device  experience  was  marginally 
significant.  Thus,  it  is  clear  from  these  results  that  the  menu 
condition  allows  experts  to  benefit  by  permitting  them  to  read 
only  a  few  frames. 

Choice  Patterns .  The  specific  pattern  of  frame  choices  for 
each  device  was  considered  in  terms  of  the  menu  hierarchy  for  each 
device.  The  intended  organization  of  the  menu  instructions  was 
that  the  levels  in  the  hierarchy  would  correspond  to  the  natural 
chunks  in  the  operation  of  the  device.  However,  contrary  to  the 
goals  of  the  experiment,  the  evidence  to  support  this  claim  is 
very  limited  in  these  data.  In  order  for  there  to  be  natural 
chunks  in  the  operation  of  the  device,  the  device  must  be  familiar 
to  the  subject.  However,  if  the  device  was  fairly  familiar  to  the 
subject,  the  subject  would  need  to  read  very  few  frames,  often 
only  the  main  task  statement  frame,  and  thus  there  would  be  few 
choices  to  reveal  which  portions  of  the  menu  hierarchy  were 
familiar  and  which  were  not.  Perhaps  different  devices  would  have 
yielded  more  useful  data. 

However,  there  were  some  interesting  patterns  in  the  choices. 
Figure  1  illustrates  the  best  example.  The  figure  shows  the  menu 
hierarchy  for  the  phi  demonstrator  in  simplified  form.  The 
terminal  portions  of  the  tree  consist  of  the  sequence  of  actual 
steps  that  were  identical  to  the  step-by-step  instructions.  In 
each  box  is  shown  the  proportions  of  nonexperts  and  experts  who 
read  the  material  in  the  box.  Thus,  for  example,  the  top-level 
box  corresponds  to  the  frame  that  states  the  main  task.  Almost 
all  subjects  then  read  the  main  menu  which  contains  four  items: 
powering  up  the  device,  attaching  the  lights,  setting  the  mode, 
and  adjusting  the  CPS  dial.  However,  only  40$  of  the  nonexperts 
and  only  10$  of  the  experts  felt  it  was  necessary  to  get  the  more 
specific  information  about  powering  up  the  device,  and  almost  none 
of  the  subjects  required  the  step-by-step  instructions  about  how 
to  plug  in  the  device  and  turn  it  on.  The  other  devices  that  also 
involved  these  steps  also  had  this  general  pattern.  Very  few 
subjects,  even  nonexperts,  required  the  specific  instructions  on 
plugging  in  and  turning  on  the  device.  This  was  true  for  the 
oscilloscope  and  signal  generator  combination,  and  also  true  for 
the  stimulator,  which  was  a  very  complicated  and  unfamiliar 
device . 

Another  effect  that  appears  in  Figure  1  is  the  tendency  for 
nonexperts  to  learn  while  doing  similar  activities.  Notice  how 
50$  of  the  subjects  required  the  step-by-step  instructions  for 
plugging  in  light  A,  but  only  10$  of  them  went  on  to  read  the 
instructions  for  how  to  plug  in  light  B.  A  similar  effect  appears 
in  the  oscilloscope  task,  in  which  fewer  nonexpert  subjects 
required  the  instructions  for  plugging  in  and  turning  on  the 
second  piece  of  equipment  than  for  the  first  piece  of  equipment. 
The  obvious  implication  of  this  effect  is  that  subjects  are  not 
simply  executing  these  instructions  as  they  read  them,  and  then 
forgetting  the  instruction  content  when  they  proceed  to  the  next 
instruction.  Rather,  they  seem  to  be  able  to  take  the  content  of 
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Figure  1.  Proportion  of  menu  choices  made  at  each  menu  level  for  experts  (E) 
and  nonexperts  (N)  on  the  phi  phenomenon  demonstrator. 
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one  instruction  and  generalize  it  immediately  to  apply  to  a 
similar  situation.  However,  these  results  are  too  limited  to  shed 
much  light  on  this  issue.  Further  work  is  clearly  needed. 

Step  Completion  Times 

Analysis  method .  A  regression  analysis  was  done  to  determine 
which  factors  predict  the  amount  of  time  taken  to  complete 
individual  steps  in  the  step-by-step  condition.  The  videotape 
scoring  was  used  to  eliminate  the  times  for  individual  steps  that 
were  defective.  In  addition,  the  times  on  the  very  first  step  in 
the  instructions  were  not  included  since  in  some  cases,  these 
times  were  contaminated  as  described  above.  This  left  a  total  of 
3008  individual  step  times  for  the  analysis.  Each  instruction 
step  was  classified  according  to  a  set  of  categories,  shown  in 
Table  10,  which  are  the  general  types  of  actions  stated  by  the 
instructions.  These  categories  were  each  represented  by  a  dummy 
variable,  with  the  ISIMP  category  being  used  as  the  baseline.  The 
video  tapes  for  each  subject  were  scored  according  to  the  action 
actually  carried  out  by  the  subject  on  each  step.  The  scoring 
categories  for  the  actions  are  shown  in  Table  1 1 .  These  were  also 
represented  with  dummy  variables,  with  SKIP  being  used  as  a 
baseline.  In  order  to  examine  the  chunking  properties  of  the 
step-by-step  instructions,  the  variable  MENU  was  defined,  which 
reflects  the  proportion  of  times  the  subjects  in  the  menu 
condition  read  the  corresponding  step.  This  variable  took  on  a 
value  that  depended  on  whether  the  subject  was  an  expert  or  a 
nonexpert.  If  the  subject  was  an  expert,  then  the  value  of  MENU 
was  the  proportion  of  experts  that  viewed  the  corresponding  step 
in  the  menu  condition.  Likewise,  for  a  nonexpert,  the  MENU 
variable  was  the  proportion  of  nonexperts  that  viewed  that 
instruction. 

An  additional  variable  that  reflected  properties  of  the 
instructions  was  the  number  of  words  in  each  instruction.  This 
variable  should  not  be  taken  to  reflect  comprehension  time,  since 
its  coefficient  is  far  too  large;  rather,  it  provides  a  crude 
measure  of  the  overall  amount  of  information  in  the  instruction. 
Additional  variables  entered  into  the  analysis  were  the  subject's 
expertise  group,  and  the  device  experience  variable,  as  described 
above.  The  subject  expertise  variable  was  allowed  to  interact 
with  all  of  the  instruction  characteristic  variables  and  the 
action  variables.  As  before,  the  order  of  entry  in  the  stepwise 
analysis  was  hierarchical,  and  the  conservative  "F-to-remove" 
statistic  is  reported.  Finally,  since  this  was  a  mixed  design, 
the  subject  expertise  variable  was  entered  into  the  equation 
first,  followed  by  a  subject  mean  variable,  then  by  the 
within-sub j ects  variables. 

Step  time  results .  A  summary  of  the  analysis  is  shown  in 
Table  1  2 .  Note  that  the  coefficients  must  be  interpreted  in  terms 
of  the  fact  that  all  other  factors  are  in  the  equation.  There  was 
a  substantial  effect  of  subject  expertise  (SUBEXP),  in  which 
experts  read  the  instructions  on  the  order  of  1.6  seconds  faster 
per  step  than  nonexperts.  Also,  the  step  times  differed 


Table  10 

Dummy  Variables  Used  to  Code  Instruction  Contents 


Variable  Description  and  Example 


ILOC 

IADJ 

ISIMP 

IEPPECT 

ICOMPH 

IEXPH 


Locate  a  part  of  the  device  (locate  the  power  switch) 
Setting  a  control  (turning  knob  to  DC) 

A  simple  action  (flipping  a  switch) 

Adjusting  a  knob  to  produce  a  certain  effect 
(zeroing  ohms  scale) 

A  complex  physical  action  (plugging  in  a  cord) 

A  complex  physical  action  familiar  to  an  expert 
(zeroing  a  meter) 


Table  1 1 

Dummy  Variables  Used  to  Code  Subject's  Actions 


Variables  Description 


DO 

SKIP 

LOOK 

LOC 


Action  same  as  instruction 
No  action  carried  out 
Subject  looks  at  device 
Subject  "locates"  a  part  of  device 
(e.g.  touches  it) 

Subject  engages  in  some  action  other  than  above 


ACT 


Table  12 

Regression  Analysis  on 
Completion  Times  for  each  Step  in 
the  Step-by-step  Condition 


Variable 

Coefficient 

Std.  Coef. 

F-to-Remove 

CONSTANT 

-15.45 

SUBEXP 

-1  .65 

-.110 

36.96 

SMEAN 

1  .00 

.249 

242.29 

DO 

4.11 

.277 

9-54 

LOC 

2.46 

.139 

4.55 

ACT 

7.30 

.188 

1 29.20 

ILOC 

1  .86 

.126 

5-43 

I ADJ 

4. 60 

.116 

45-99 

IEFFECT 

3. 1  6 

•  093 

18. 92 

WORDS 

•  57 

•  339 

366.98 

ICOMPH 

14.46 

.247 

154.99 

IEXPH 

6.42 

.156 

42.66 

EXPCOM 

-5.52 

-.065 

11.19 

EXPXPH 

-5-78 

-.100 

24.31 

MENU 

1  .71 

.091 

8.59 

Notes 

R~2  is  .4075  with  19  variables  and  N=3008.  Five  variables 
are  not  shown  because  the  F-ratios  were  nonsignificant.  See  text 
for  explanation  of  variables.  Values  for  SUBEXP  are  based  on  only 
SUBEXP  in  the  equation,  before  SMEAN  and  the  within-subjects 
variables  are  added. 
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substantially  depending  both  on  which  actions  that  subjects 
actually  performed,  and  also  in  the  properties  of  the  instructions 
themselves.  This  result  in  itself  is  not  too  surprising. 
However,  it  is  noteworthy  that  two  of  the  strongest  (as  shown  by 
the  standardized  regression  coefficients)  instruction  factors  are 
the  number  of  words  in  the  instruction  (WORDS),  and  whether  the 
instruction  required  a  complicated  physical  activity  (ICOMPH). 
Instructions  that  required  physical  activities  that  are  familiar 
only  to  experts,  such  as  adjusting  the  zero  adjust  screw  on  the 
VOM  (IEXPH),  also  took  significantly  longer,  even  though  such 
cases  were  fairly  rare. 

The  key  results  are  the  interactions  of  expertise  with  two  of 
the  instruction  characteristics,  namely,  complicated  physical 
activities  (EXPCOM) ,  and  expert  physical  activities  (EXPXPH). 
This  suggests  that  not  only  are  experts  faster  across  the  board, 
but  they  are  especially  fast  at  certain  complicated  physical 
activities.  Informal  observation  of  the  video  tapes  seems  to 
confirm  this.  Nonexpert  subjects  often  spend  a  lot  of  time 
fumbling  with  cords  and  connectors,  while  experts  seem  to  know 
exactly  what  they  are  doing  in  these  physical  activities,  and 
proceed  smoothly  and  precisely. 

An  additional  key  result  is  that  the  MENU  variable  was 
significant.  The  coefficient  means  that  with  all  other  factors  in 
the  equation,  a  step  that  was  always  read  in  the  menu  condition 
took  about  1.7  seconds  longer  than  one  that  was  never  read. 
Assuming  that  the  menu  choices  reflect  the  familiarity  of 
procedure  "chunks,"  the  amount  of  time  taken  to  complete  a  step  is 
thus  a  function  of  its  predictability  on  the  basis  of  prior 
knowledge . 

Knowledge-based  Operation 

In  Kieras  (1982)  it  was  proposed  that  people's  knowledge  of 
electronic  devices  is  organized  as  a  hierarchy  of  schemas,  which 
would  contain,  among  other  things,  schematic  information  on  how  to 
operate  the  corresponding  class  of  devices.  It  is  natural  to 
suppose  that  just  as  a  story  schema  specifies  the  order  of 
appearance  of  items  in  a  story,  that  a  device  schema  would  specify 
the  order  of  the  steps  for  operating  the  device.  Thus,  when 
subjects  operate  a  device  based  only  on  their  knowledge,  there 
would  be  a  stereotyped  sequence  of  behavior  corresponding  to  the 
procedural  schema  for  operating  the  device.  Some  of  the  data  from 
the  menu  condition  was  suitable  for  examining  this  issue;  there 
were  many  cases  where  subjects  attempted  to  operate  the  device 
after  receiving  only  the  main  task  statement,  without  requesting 
further  instructions. 

Analysis  method .  The  videotape  record  of  the  subjects' 
behavior  was  scored  in  terms  of  the  individual  activities  that 
subjects  performed,  such  as  operating  a  certain  control.  Data 
were  dropped  for  subjects  who  got  confused  in  the  task  or  did  it 
incorrectly  in  some  way  that  would  invalidate  the  data.  Both 
experts  and  nonexperts  all  operated  the  radio  and  cassette 


recorder  without  any  further  instructions  in  the  menu  condition. 
Seven  experts  on  the  radio  and  nine  nonexperts  were  thus 
available.  For  the  cassette  recorder  there  were  eight  usable 
behavior  sequences  from  each  group.  With  three  other  devices, 
only  experts  operated  the  device  without  instructions.  For  the 
VOM,  oscilloscope  and  signal  generator  combination,  and  the  phi 
phenomenon  demonstrator,  there  were  five,  eight,  and  eight  such 
sub j ects . 

The  method  of  analyzing  this  sequence  data  was  to  locate 
sequences  of  activities  that  occurred  at  least  twice,  and  then 
express  the  sequences  that  subjects  performed  with  as  few  terms  as 
possible  by  referring  to  these  common  sequences.  More 
specifically,  the  sequence  data  was  represented  as  a  transition 
network  tree  diagram,  in  which  the  nodes  represent  either 
individual  actions  or  action  "subroutines,"  and  a  single  path 
through  the  tree  diagram  represents  the  activities  of  a  single 
subject.  See  Figure  2  for  an  example.  Each  action  is  represented 
by  a  two-letter  symbol,  and  action  subroutines  by  combinations  of 
these  symbols.  The  depth  of  combination  is  indicated  by  the 
notation;  concatenated  symbols  are  the  shallowest  level,  with 
brackets  and  parentheses  indicating  deeper  levels  of  subroutines. 

In  order  to  construct  this  transition  diagram,  all  actions 
except  specific  control  operations  were  deleted  from  the  behavior 
stream.  Thus,  for  example,  activities  of  locating  (touching)  a 
control,  or  looking  at  various  parts  of  the  device  were  dropped 
from  the  analysis.  The  resulting  sequences  were  then  subjected  to 
a  sorting  process  in  which  common  sequences  were  identified  and 
then  the  data  regrouped  according  to  the  sequences,  and  the 
process  repeated  until  no  more  sequences  could  be  formed. 

Once  these  sequences  were  defined,  the  behavior  patterns  for 
all  of  the  subjects  could  be  rewritten  as  a  tree  diagram,  in  which 
all  subjects  begin  at  the  origin  and  then  branch  out  according  to 
the  first  action  or  sequence  subroutine  that  they  perform,  and 
then  branch  out  further  depending  on  their  individual  actions. 
Since  all  subjects  eventually  did  some  action  that  was  different 
from  that  done  by  any  other  subject,  eventually  the  trees  all  had 
the  same  number  of  branches  as  there  were  subjects. 

Pat  tern  results .  In  Figure  2  is  shown  the  top  level  diagram 
for  the  sequences  for  the  nonexperts  and  experts  on  the  radio. 
Notice  how  the  nonexpert  network  seems  to  be  "bushier"  than  the 
expert  network,  and  also  appears  to  have  more  different 
subroutines.  Beyond  the  preference  for  initially  plugging  in  and 
turning  on  the  radio,  there  seems  to  be  little  in  the  way  of  an 
interpretable  pattern  in  the  nonexpert  sequences.  However,  there 
is  a  basic  pattern  to  the  expert  sequences.  The  subjects  who 
followed  the  bottom  two  major  branches  first  "set  up"  some  portion 
of  the  radio  before  turning  it  on.  The  subjects  following  the 
upper  branch  turned  on  the  radio  immediately  and  then  proceeded  to 
make  a  series  of  adjustments  to  it.  Thus,  even  with  as  simple  a 
device  as  a  radio,  there  seem  to  be  two  major  methods  of  operating 
it:  the  first  is  setting  it  up  and  then  turning  it  on,  followed 


Figure  2.  Behavior  sequences  for  expert  and  nonexpert  subjects 
operating  the  radio  without  instructions. 
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by  adjusting  it,  and  the  second  is  turning  it  on,  then  setting  it 
up  and  adjusting  it.  Within  each  of  these  two  major  patterns 
there  are  many  minor  variations. 

A  similar  apparent  difference  between  experts  and  nonexperts 
appears  with  the  cassette  recorder  in  Figure  Overall,  the 
experts  appear  to  produce  shorter  and  simpler  sequences  than  the 
nonexperts.  Thun  the  experts  in  both  the  radio  and  the  tape 
recorder  appear  to  have  more  consistent  and  shorter  behavior 
sequences.  Some  quantitative  comparisons  between  the  expert  and 
nonexpert  transition  networks  were  very  intriguing,  but  none  of 
them  reached  statistical  significance. 

It  should  be  noted  that  some  of  the  complexity  of  the  tape 
recorder  behavior  sequences  is  probably  due  to  the  fact  that  the 
tape  cassette  was  deliberately  given  to  the  subjects  in  a 
condition  in  which  it  was  not  fully  rewound.  Since  the  subjects' 
task  was  to  record  "testing  one-two-three"  on  the  tape  and  play  it 
back,  this  confused  some  subjects  if  they  rewound  the  tape  all  the 
way  back  after  recording  as  one  normally  would.  Thus,  some 
subjects,  even  experts,  had  to  make  more  than  one  attempt  to 
record  the  tape.  Perhaps  this  complexity  is  a  reflection  of  the 
fact  that  the  tape  recorder  was  not  left  in  a  schematic  state; 
that  is,  the  normal  state  for  a  tape  cassette  is  that  it  is  fully 
rewound . 

An  important  conclusion  is  that  if  there  is  an  apparent 
difference  between  experts  and  nonexperts,  even  on  these  everyday 
devices,  then  experts  are  better  even  at  operating  everyday 
devices  than  nonexperts.  This  presents  a  serious  problem  for 
future  studies  of  electronics  expertise,  because  it  suggests  very 
strongly  that  nonexperts  can  not  be  used  as  subjects  of  such 
studies  even  if  very  familiar  devices  are  used. 

A  further  result  that  follows  from  an  examination  of  these 
two  networks,  and  was  also  clearly  apparent  with  the  other  devices 
i3  that  there  is  in  fact  very  little  stereotypy  in  the  specific 
behavior  sequences.  Figure  4  presents  the  transition  network  for 
the  five  experts  using  the  VOM.  Notice  that  the  number  of 
subroutines  is  quite  small,  and  there  is  an  almost  immediate 
branching  of  the  tree  into  unique  paths,  one  for  each  subject. 

Because  of  the  extreme  length  of  the  sequences  for  the 
oscilloscope  and  signal  generator  combination,  Figure  b  presents  a 
truncated  and  condensed  version  of  the  full  transition  network. 
For  example,  the  terra  CRT  means  any  control  activities  having  to 
do  with  adjusting  the  CRT  trace  on  the  oscilloscope,  which  could 
involve  any  sequence  of  the  five  controls.  Likewise,  TB  refers  to 
any  sequence  involving  adjustments  to  the  oscilloscope's  time 
base,  which  also  involved  several  controls.  It  should  be  noted 
that  even  after  this  extreme  condensation  the  paths  through  the 
network  again  branch  into  unique  patterns  very  quickly.  The  phi 
phenomenon  demonstrator  in  Figure  6  also  shows  a  relatively  quick 
branching  into  unique  paths. 
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Lack  of  fixed  procedures .  The  fact  that  on  the  whole  there 
is  very  Tittle  stereotyped  behavior  see;  s  to  disconfirm  the 
hypothesis  suggested  above,  whicn  is  that  device  schema  knowledge 
tightly  specifies  operating  procedures  for  devices.  However,  it 
should  be  pointed  out  that  there  are  some  strong  consistencies  in 
at  least  the  initial  stages  of  operating  at  least  some  of  the 
devices.  For  example,  with  the  radio  (Figure  2  ),  all  subjects 
plugged  it  in  first.  With  the  recorder,  roughly  half  of  the 
experts  and  nonexperts  plugged  the  device  in  as  the  first  step. 
With  the  oscilloscope,  most  of  the  subjects  plugged  in  and  turned 
on  both  the  oscilloscope  and  the  signal  generator  before  going  any 
further,  but  there  were  some  subjects  that  performed  only  part  of 
this  operation  before  proceeding.  LiKewise,  notice  that  many 
subjects,  after  performing  the  power-up  operations,  went  on  to 
connect  the  two  devices  together  before  proceeding  any  further. 
Finally,  with  the  demonstrator,  again  most  of  the  subjects  plugged 
in  the  cord  first,  although  some  of  them  plug  in  all  of  the  cords 
and  connectors  before  turning  on  the  device. 

The  VOM  presents  an  interesting  contrast,  because  it  does  not 
have  to  be  plugged  in  and  turned  on.  Notice  that  there  is  very 
little  stereotypy  in  the  sequence  of  activities.  One  might  think 
that  inserting  the  test  leads  would  be  the  natural  first  step,  but 
only  two  of  the  five  subjects  did  this.  Or  one  might  think  that 
adjusting  the  meter  to  zero  would  be  a  natural  first  step;  only 
one  of  the  subjects  did  so,  although  it  should  be  noted  that  this 
is  not  a  routine  operation  in  the  normal  use  of  a  meter  of  this 
type.  Thus,  it  appears  that  there  is  some  stereotyped  behavior, 
but  it  is  limited  to  some  of  the  very  initial  stages  of  device 
operation,  and  concerns  mainly  "power-up"  procedures.  If  people 
indeed  follow  schematic  procedures,  these  procedures  are  of  such  a 
limited  and  varied  nature  that  characterizing  them  as  schemas  is 
of  little  value. 

How  subjects  operate  from  memory .  This  lack  of  stereotypy 
requires  explanation.  Closer  examination  of  the  task  situation  of 
operating  a  device  from  memory  suggests  that  the  expectation  that 
device  operation  would  show  stereotyped  orders  is  not  reasonable. 
That  is,  although  the  devices  were  representatives  of  a  very 
familiar  type  of  device,  such  as  a  radio,  the  likelihood  that  an 
individual  subject  had  actually  had  extensive  practice  with 
operating  this  particular  make  and  model  of  device  is  essentially 
zero.  To  some  extent,  every  device  was  a  novel  device  to  every 
subject.  Thus,  none  of  the  actual  skills  of  operating  the  device 
would  be  highly  automated,  because  this  would  only  be  the  case  if 
one  were  familiar  with  the  specific  location  and  properties  of  the 
particular  device.  Thus,  subjects  were  essentially  operating 
these  devices  in  a  problem-solving  mode,  instead  of  a  memory 
retrieval  mode.  Once  the  problem  is  looked  at  in  this  light,  the 
lack  of  stereotypy  in  the  behavior  becomes  clear. 

In  any  actual  device,  there  are  constraints  that  are  imposed 
by  the  device  on  the  order  in  which  things  are  done.  For  example, 
on  an  oscilloscope,  the  intensity  control  must  be  adjusted  before 
a  trace  can  be  seen,  and  the  oscilloscope  can  not  be  used  until 
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the  trace  is  visible.  Before  the  intensity  control  can  be 
properly  adjusted,  however,  the  oscilloscope  must  be  turned  on. 
Thus,  for  any  device  there  are  some  constraints  on  the  order  of 
certain  operations.  However,  even  for  relatively  simple  devices, 
such  as  a  radio,  these  constraints  in  fact  specify  very  little  of 
the  exact  order  of  operation;  many  steps  are  independent  of  order 
given  that  the  overall  constraints  are  met.  For  example,  the 
radio  tuning  can  be  adjusted  at  any  time,  but  most  usefully  after 
the  radio  is  audibly  playing.  Thus,  referring  to  Figure  2,  there 
are  many  different  orders  in  which  the  expert  subjects  operated 
the  controls  on  the  radio,  and  there  is  a  unique  path  for  every 
subject.  However,  all  of  the  subjects  succeeded  in  operating  the 
radio,  and  typically  with  very  little  wasted  time  or  steps. 

Conclusion .  The  best  characterization  of  operating  a  piece 
of  equipment  from  memory  seems  to  be  that  subjects  perform 
problem-solving  by  determining  what  constraints  need  to  be 
satisfied  along  the  way,  and  then  operating  the  controls  in  a 
manner  that  meets  the  constraints  and  accomplishes  the  task,  but 
does  not  necessarily  follow  any  prescribed  order.  Since  a  major 
constraint  is  that  the  device  be  operating  before  it  can  be 
adjusted,  there  is  a  strong  tendency  for  "power-up"  steps  to  be 
done  first.  Since  these  data  involve  only  a  single  observation  on 
each  subject  in  each  device,  it  is  impossible  to  tell  whether  each 
subject  was  following  an  individual  stereotyped  sequence,  which 
seems  unlikely.  However,  it  is  very  clear  that  device  operating 
sequences  do  not  have  a  major  property  of  schemas,  namely, 
stereotypy  of  content. 

The  larger  implication  of  this  conclusion  is  that  even  though 
experts  can  operate  even  complex  pieces  of  equipment  completely 
from  prior  knowledge,  they  do  not  perform  this  by  rote  memory 
retrieval  but  rather  by  a  very  general  problem-solving  process. 
For  example,  the  best  characterization  of  what  the  experts  did 
with  the  oscilloscope  is  that  once  they  had  it  plugged  in,  turned 
on  and  connected  with  a  signal  generator,  they  made  many  passes 
over  the  controls  making  various  fine  adjustments  in  all  sections 
of  the  oscilloscope  until  they  had  achieved  the  final  desired 
result.  Many  of  the  operations  were  undoubtedly  redundant  from  a 
strictly  technical  point  of  view.  However,  these  general 
processes  are  powerful  enough  that  the  experts  could  operate  the 
completely  novel  device,  the  phi  phenomenon  demonstrator,  without 
any  instructions,  and  quite  often  without  any  serious  mistakes  or 
wasted  actions. 

The  general  conclusion  is  that  expertise  does  not  consist  of 
a  set  of  canned  procedures  for  operating  different  devices,  but 
rather  of  a  set  of  powerful  problem-solving  heuristics  which  can 
be  applied  to  even  novel  devices,  but  which  are  not  very  efficient 
even  with  familiar  devices. 
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SUMMARY 

The  introduction  listed  three  questions  that  this  experiment 
was  designed  to  address.  These  concerned  the  instruction  format, 
the  nature  of  expertise  effects,  and  the  nature  of  the  prior 
knowledge  that  people  would  have  about  electronic  equipment.  This 
experiment  yielded  information  about  each  of  these  three  questions 
which  can  be  summarized  as  follows: 

Instruct i on  Format.  Contrary  to  intuition,  the  menu  format 
was  not  better  overall  than  the  step-by-step  format;  which  format 
is  superior  depends  on  the  user's  experience.  Under  some 
conditions  the  specific  experience  with  the  actual  device  involved 
can  be  more  important  than  the  user's  general  expertise.  If  the 
device  is  faailiar,  the  menu  format  helps,  as  would  be  expected, 
by  reducing  the  amount  of  instructions  that  must  be  read. 
Subjects  tend  not  to  read  familiar  steps  such  as  descriptions  of 
how  to  power-up  the  equipment  which  everyone  knows,  nor  do  they 
read  descriptions  of  procedures  that  are  very  similar  to  ones  they 
have  just  completed.  If  a  device  is  not  familiar,  the  user  can  go 
astray,  and  the  result  may  be  much  worse  than  using  step-by-step 
instructions  in  terms  of  total  completion  time. 

Expertise  Effects .  Expertise  had  both  specific  and  general 
effects  in  tnese  results.  Experts  were  faster  overall,  both  in 
the  menu  and  tne  step-by-step  conditions.  But  experience  with  the 
specific  device  can  be  as  important  as  the  general  experience. 
The  experts  were  more  efficient  than  the  nonexperts  in  terms  of 
being  able  to  carry  out  complicated  physical  activities.  Although 
everybody  know  3  certain  things  about  electronic  equipment,  such  as 
how  to  turn  on  a  device,  even  on  everyday  devices  the  experts  are 
more  efficient  and  more  consistent  in  their  activities  than 
nonexperts . 

Prior  knowledge  of  electronic  devices .  It  was  proposed  that 
since  people  apparently  have  schema  knowledge  for  electronic 
devices,  that  they  would  also  have  knowledge  of  schematic 
procedures  for  operating  devices.  A  primary  characteristic  of 
such  schematic  procedures  would  be  a  high  degree  of 
stereotypicality  in  how  the  devices  were  operated  when  subjects 
did  not  choose  to  read  instructions.  This  expectation  was 
contradicted  oy  the  data;  there  was  very  little  stereotyped 
behavior  when  subjects  operated  the  devices  strictly  on  the  basis 
of  their  prior  knowledge. 

A  more  accurate  assessment  is  based  on  making  a  distinction 
between  what  people  do  when  they  have  a  highly  automated  skill  at 
operating  a  particular  piece  of  equipment,  and  the  ability  to 
operate  equipment  in  a  more  normal  setting  in  which  every  piece  of 
equipment  is  familiar,  but  not  highly  practiced.  In  this  case, 
what  subjects  do  is  to  engage  in  complicated  problem-solving 
strategies,  where  the  individual  operating  steps  meet  loose 
constraints  that  are  imposed  by  the  nature  of  the  device,  but  do 
not  otherwise  fall  into  a  strict  stereotyped  sequence.  This 
problem-solving  strategy  is  very  robust  but  it  is  inconsistent 


Page  1  9 


between  individuals  and  can  be  inefficient.  Experts  clearly  have 
much  more  powerful  strategies  than  nonexperts  for  operating 
devices  on  the  basis  only  of  prior  knowledge,  but  in  the  case  of 
unfamiliar  equipment,  their  performance  may  actually  be 
considerably  poorer  than  that  of  nonexperts  who  are  following 
strict  step-by-step  instructions . 
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